Joint Science Operations Center

Status:

• All resources from the Stanford JSOC website and data system are unavailable until repairs and recovery are completed
• IRIS Level 2 data through November 22, 2024 remain available at iris.lmsal.com/search/
• The Virtual Solar Observatory is the best location to find historical AIA and HMI science data through November 26, 2024, though not all dates and products are available
• SDO EVE data products are not impacted by this outage

JSOC Recovery Updates

The JSOC team is restoring the SDO data analysis pipeline as quickly as possible and has made significant progress in the past two weeks. We are also working to restore external access to the SDO data archive, but that capability remains unavailable. Some current observations are now available at sdo.gsfc.nasa.gov, suntoday.lmsal.com, and jsoc1.stanford.edu/hmi_latest.html

We have begun processing most near-real-time (nrt) SDO data products. The primary exception is the SHARPs, which require sequential processing. We are in the process of back-filling so that SHARP processing can resume. We have been producing definitive Level 0 and Level 1 data for HMI and AIA as the calibrations become available. We are also beginning to make definitive observables (Level 1.5 science images) for both HMI and AIA. These data are being processed from the time of the flooding incident going forward.

The database server is functioning adequately for the purpose of the pipeline, and "warm" back-up database servers have been enabled. However, we are expecting a few-week delay before bringing that part of the system up to full functionality as we wait for replacement hardware. Until then, we will not be able to provide access to the archived SDO data. We are working on alternative data-access methods; watch for updates. Replacements for the damaged data servers are expected to arrive soon, which will allow us to more fully evaluate storage disks and restore the online data sets.

All routine IRIS data pipeline products from JSOC are being produced regularly. A few intervals are awaiting the arrival of housekeeping data that was not processed at the time.

Some progress to report. As of 19 December 2024, the team was able to restart the first stage of the processing pipeline for incoming data from the AIA and HMI instruments and the IRIS mission. This stage creates and archives Level 0 images (uncalibrated raw images). Level 1 processing may be able to resume when the backlog of Level 0 processing is cleared, but the JSOC data system is operating at significantly reduced capacity while we await the delivery of replacement hardware.

Replacement hardware has been ordered, and a few components have arrived, but delivery dates for major components such as the database servers and disk servers are uncertain over the winter break.

We do not yet have a timeline for resumption of higher level processing and export to the community. In parallel with our recovery efforts we are working on alternative options for data access, but unfortunately do not yet have any solutions in place.

The instruments are healthy and continue to operate normally; no science data has been lost. We continue to be thankful for your patience and understanding.

As of 9 December 2024 the assessment of the damage to the JSOC due to the 26 November flood is continuing and replacements of equipment are commencing. Restoration of the functioning system will come after repairs to the database hardware are complete.

First some good news.

• The JSOC facility is now dry.
• The data capture systems for HMI, AIA, and IRIS continue to operate, and we have implemented a temporary additional offsite back-up of the incoming telemetry packets at the GSFC SDAC, so we do not expect that any data will be lost.
• The tape drive library system appears undamaged, as do the on-site tape archives that hold the rest of the JSOC data. This will make recovery of the historic data much faster.
• We are now able to produce level 0 image data suitable for monitoring the basic performance of the instrument cameras.
• We are in the process of ordering replacements for damaged equipment. We do not yet have a schedule for delivery. The Stanford University administration has been very helpful about minimizing the delays in ordering replacements.

Now for some not so good news.

• Water damage was concentrated in the area where the leaking water pipe sprayed water floor to ceiling. The rest of the JSOC floor was under a few inches of water, but most equipment was raised above the floor level. Other laboratories in the building suffered less impact, though water flowed into the halls, under doors, and down to the subbasement and damaged the elevator.
• Several (roughly 20%) of the database servers and data servers were damaged. Several large data partitions that are on disk do not function. Quite a number of disk drives were damaged; however, assessment of how much we expect to be able to recover from those is ongoing. Several UPS power supplies were damaged, along with a substantial inventory of unused tapes. A number of other computer systems, including our web servers and the WSO data reduction computer, were also damaged.

Plans for the coming weeks

Restoration of the database servers that manage the storage of the billions of files in the JSOC repository is our first and foremost challenge. This is our highest priority, since other elements of the recovery depend on having a fully functional and secure data base management system. Even though we do have a partially functioning system that helps assess the degree of damage and identify what is missing, the restoration will take time to repopulate both the data base and the data archives. The JSOC is not presently able to produce calibrated images or higher level data products. Restarting the data processing pipeline depends on having a functioning database management system to access ancillary calibration and housekeeping information and to organize new data products as they are made.

We are grateful for your patience as our small team works to restore the JSOC functionality. Continue to check our JSOC Emergency Resources page for alternative ways to access some HMI and AIA data in the mean time.

On Tuesday, November 26, 2024 a 4-inch chilled-water pipe in the Solar Dynamics Observatory (SDO) Joint Science Operations Center (JSOC) server room broke. This caused major flooding in the building and extensive water damage in the lab that houses the machines that process and distribute data from the Helioseismic and Magnetic Imager (HMI) and Atmospheric Imaging Array (AIA) instruments and from the IRIS spacecraft. The Stanford JSOC team is working to assess the extent of the damage, but it is severe. Science data processing for HMI, AIA, and IRIS will be down for an extended length of time, as will access to the archived data at JSOC.

The data capture systems for all three instruments remain functional, so ultimately incoming data are not expected to be lost. Instrument health and safety monitoring is being handled at the Instrument Operations Center at the Lockheed Martin Solar & Astrophysics Lab (LMSAL). The instruments continue to function normally.

The team will provide updates as more information becomes available. We deeply regret any inconvenience and appreciate your patience as our team prioritizes the repair and recovery of the affected systems.